Modeling The Language Assessment Process And Result: Proposed Architecture For Automatic Oral Proficiency Assessment

We outline challenges for modeling human language assessment in automatic systems, both in terms of the process and the reliability of the result. We propose an architecture for a system to evaluate learners of Spanish via the Computerized Oral Proficiency Instrument, to determine whether they have 'reached' or 'not reached' the Intermediate Low level of proficiency, according to the American Council on the Teaching of Foreign Languages (ACTFL) Speaking Proficiency Guidelines. Our system divides the acoustic and non-acoustic features, incorporating human process modeling where permit ted by the technology and required by the domain. We suggest machine learning techniques applied to this type of system permit insight into yet unarticulated aspects of the human rating process. 1 I n t r o d u c t i o n Computer-mediated language assessment appeals to educators and language evaluators because it has the potential for making language assessment widely available with minimal human effort and limited expense. Fairly robust results (n '~ 0.8) have been achieved in the commercial domain modeling the human rater results, with both the Electronic Essay Rater (erater) system for written essay scoring (Burstein et al., 1998), and the PhonePass pronunciation assessment (Ordinate, 1998). There are at least three reasons why it is not possible to model the human rating process. First, there is a mismatch between what the technology is able to handle and what people manipulate, especially in the assessment of speech features. Second, we lack a wellarticulated model of the human process, often characterized as holistic. Certain assessment features have been identified, but their relative importance is not clear. Furthermore, unlike automatic assessments, human raters of oral proficiency exams are trained to focus on competencies, which are difficult to enumerate. In contrast, automatic assessments of spoken language fluency typically use some type of error counting, comparing duration, silence, speaking rate and pronunciation mismatches with native speaker models. There is, therefore, a basic tension within the field of computer-mediated language assessment, between modeling the assessment process of human raters or achieving comparable, consistent assessments, perhaps through different means. Neither extreme is entirely satisfactory. A spoken assessment system that achieves human-comparable performance based only, for example, on the proportion of silence in an utterance would seem not to be capturing a number of critical elements of language competence, regardless of how accurate the assessments are. Such a system would also be severely limited in its ability to provide constructive feedback to language learners or teachers. The e-rater system has received similar criticism for basing essay assessments on a number of largely lexical features, rather than on a deeper, more humanstyle rating process. Thirdly, however, even if we could articulate and model human performance, it is not clear that we want to model all aspects of the human rating process. For example, human performance varies due to fatigue. Transcribers often inadvertently correct examinees' errors of omitted or incorrect articles, conjugations, or affixes. These mistakes are a natural effect of a cooperative listener; however, they result in an over-optimistic assessment of the speaker's actual proficiency. We arguably do not wish to build this sort of cooperation into an automated